User-Centered Music Information Retrieval Evaluation

Evaluation in Music Information Retrieval (MIR) has been focused on system-centered approaches where systems are evaluated against pre-built ground truth datasets using objective measurements without taking users into account. For example, no user interactions are considered in the various tasks held in the Music Information Retrieval Evaluation eXchange (MIREX), the most influential, community-based evaluation framework in MIR. In this paper, we propose a user-centered MIR evaluation paradigm by introducing and adopting the measures from the established area of Interactive Information Retrieval in the text domain. The purpose is to inspire and encourage MIR researchers to adopt user-oriented approaches in evaluating MIR systems.